The present invention relates to the retrieval and caching of data from a storage area network and in particular to a device adapter that enables the caching of data accessed from a storage without the support or alteration of an operating system (OS).
Network access storage systems, such as RAIDs (Redundant Arrays of Inexpensive Disks), that are accessed through a SAN (Storage Area Network) are extremely useful in terms of reliability and storage capacity expandability, and enable extremely fast data input and output (I/O). Such network access storage systems can be accessed with adaptability using fiber channel (FC) networks and IP networks to realize SANs and by using standard SCSI commands and SCSI/IP protocols such as iSCSI. As a result, multiple storage devices connected to a SAN can be simultaneously directly accessed from multiple host computers.
In such network access storage systems, the input/output speed may be restricted and the performance of the overall system may be diminished due to data of a storage device connected to a SAN being simultaneously accessed by multiple host computers. Specifically, the rate at which data is transferred from the storage device to the host computers is restricted by the physical bandwidth of the network media.
Thus, in some cases, even if the storage device can provide data rapidly, the system can still be slower than desired. Switches configured in the SAN process data transferred between multiple ports, either from the host computers to the storage device or from the storage device to the host computers. For that reason, a wide input/output internal bandwidth is required and it is necessary to efficiently process all input/output operations.
Two problems arise in the host computer accessing data through the network connected to the storage system, a drop in the data transfer rate and an increase in the amount of time necessary for input/output request processing. If the host computer is used as a multimedia streaming server, these problems are particularly serious because the performance of the server is restricted by its ability to retrieve data from the storage system to send the data to the client.
Specifically, if the data retrieval rate is slow, the host computer's performance as a server drops regardless of the operational processing ability of the host computer. When consideration is given to the sending of multimedia data requiring real time processing, data transmission delays arise due to the increased processing time for input/output requests. Thus, the quality of the multimedia that is sent, decoded and provided to the client may be impaired. Many storage systems designers are wrestling with these performance problems, and different solutions are being proposed to eliminate these.
First, solutions are being engineered in many storage systems by disposing a fast cache memory to circumvent access delays of the storage medium (e.g., a magnetic disk or tape) and to overcome delays in the input/output processing of the storage system. With this cache memory, caching of frequently accessed data becomes possible and lower-speed storage media may be used, while the amount of processing time of a request to input/output data is reduced (e.g., see Japanese Laid Open Patent Publication JP-A-2001-051901).
Caching methods in which the data are cached near a host computer are better solutions because the amount of consumption of the data transfer bandwidth inside the SAN can be reduced. Several cache devices connected to a device adapter used to access a SAN or a host expansion bus have been proposed. For example, in the device adapter disclosed in U.S. Pat. No. 6,463,509, a cache device is directly connected to an expansion bus (I/O bus) of a host computer. In this method, to cache a large volume of data, extremely flexible caching is realized by using a disk of an extremely large capacity as a cache storage medium.
There is also a method where caching operations are conducted in a device that is connected to a storage system and used to retrieve data from a SAN. Although the usefulness of the caching of data in the storage accessed by this device is reduced, overload of the host expansion bus during caching can be relieved.
As an example of this kind of device, the device adapter disclosed in International Publication No. 03/017598 can cache, in a disk connected to an access device, data retrieved from a storage system connected to an IP network. Thus, caching operations can be implemented in the device adapter without overloading the host expansion bus.
As an example of a host computer connected to a data cache, caching methods have been proposed that do not give consideration to differences in processing speeds between accessed storage systems and the highest data transfer rates of used cache storage. In the method disclosed in JP-A-2001-051901, however, use of the network bandwidth in the SAN cannot be reduced because caching is conducted in the access-destination storage device. Thus, this method cannot solve the drop in data transfer performance in the SAN. Moreover, ordinarily the size of the cache is not a size sufficient to retain a large amount of data, and with multimedia data files accessed by a streaming server, the efficiency of the overall cache drops in a case where multiple host servers continuously and simultaneously access large-volume data files.
In the method disclosed in U.S. Pat. No. 6,463,509, data retrieved from various storage devices through different types of interfaces can be cached in a caching device of a host operating system by changing device drivers and data transfer paths in the host operating system. With this method, high flexibility is achieved, but two serious problems arise. The first is that support from the operating system becomes necessary to process data caching and retrieval from the caching device. The use target operating system must support the hierarchy of the device driver.
The other problem is that the consumption of the bandwidth of the caching process-use expansion bus becomes excessive. There is the potential for the data that is to be cached to be transferred two times by the expansion bus in accordance with the type of bus used. The first time is when the data is transferred to the host memory from the device accessing the storage data, and the second time is when the data is transferred to the cache device from the host memory. There is the potential for performance in a where input/output is concentrated, such as multimedia streaming, to drop due to overloading of the host I/O bus.
In the method disclosed in International Publication No. 03/017598, the device is not given the function of conducting bandwidth control to precisely measure the load of the cache storage being used and the load of the interface used to retrieve data that is not cached, and the data retrieval destination is determined depending on whether or not the data is in the cache. The function of dynamically determining the data retrieval destination using the load status of the interface is necessary in the case of the retrieval of data characterized by real time processing, such as multimedia data.
In a case where, with respect to a specific type of input/output processing, the cache device performance during use is lower than that of the SAN connected to the storage device, the overall system could potentially drop. In a case where the obtained input/output request processing time (or at least an estimation thereof) is shorter than the input/output request processing time necessary for the input/output request processing of the cache device, there is the potential for the need to retrieve data from the storage system through the SAN to arise even if the data has already been cached. A classic example is the retrieval of data of small data blocks. Retrieval of small data blocks is ordinarily faster in a disk array system, due to the effect of disk striping, and slower in a single disk system whose disk seek overhead is large.